Methods and systems for performing visual collaboration between remotely situated participants

ABSTRACT

Embodiments of the present invention are directed to a visual-collaborative systems and methods enabling geographically distributed groups to engage in face-to-face, interactive collaborative video conferences. In one aspect, a method for establishing a collaborative interaction between a local participant and one or more remote participants includes capturing images of the local participant in front of a display screen. The includes collecting depth information of the local participant located in front of the display screen and transmitting the images and depth information of the local participant to the one or more remote participants. The method also includes receiving images and depth information of the one or more remote participants and projecting the images of the one or more remote participants on the display screen based on the depth information of the remote participants.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation-in-part of application Ser. No.12/432,550 filed Apr. 29, 2009, which is a continuation-in-part ofapplication Ser. No. 12/321,996, filed Jan. 28, 2009, both of which arehereby incorporated by reference in their entirety.

TECHNICAL FIELD

Embodiments of the current invention relate to remote collaborationsystems.

BACKGROUND

Some of the most productive interactions in the workplace occur when asmall group of people get together at a blackboard or a whiteboard andactively participate in presenting and discussing ideas. However it isvery hard to support this style of interaction when participants are atdifferent locations, a situation that occurs more and more frequently asorganizations become more geographically distributed. To date,conventional video-conferencing systems are not well suited to thisscenario. Effective collaboration relies on the ability for the partiesto see each other and the shared collaboration surface, and to see wherethe others are looking and/or gesturing. Conventional video-conferencingsystems can use multi-user screen-sharing applications to provide ashared workspace, but there is a disconnect from the images of theremote participants and the cursors moving over the shared application.

FIGS. 1-3 show schematic representations of systems configured toproject images without interfering with images captured by a camera.FIG. 1 shows a communication medium with a half-silvered mirror 102, acamera 104 located above the mirror 102, and a projector 106. The mirror102 and the projector 106 are positioned so that an image of a person orobject located at a remote site is projected by the projector 106 ontothe rear surface of the half-silvered mirror 102 and is visible to aparticipant 108. The camera 104 captures an image of the participant 108via the participant's reflection in the mirror 102 and transmits theimage to another person. The configuration of mirror 102, projector 106,and camera 104 enable the participant 108 to have a virtual face-to-faceinteraction with the other person. However, close interaction betweenthe participant 108 and the other person can be disconcerting becausethe tilted screen makes for unnatural views of the remote user. FIG. 2shows a communication medium with a switchable diffusing screen 202, acamera 204, and a projector 206. The screen 202 can be composed of amaterial that can be cycled rapidly between diffusive and transparentstates. The state of the screen 202, projector 206, and camera 204 canbe synchronized so that the projector 206 projects images when thescreen is diffusive and the camera 204 captures images when the screenin transparent. However, it is difficult to design a screen that canswitch fast enough to avoid flicker, and the need to synchronize thesefast switching components adds to the complexity of the system andlimits the projected and captured light levels. FIG. 3 shows a top viewof a communication medium with two cameras 302 and 304 on each side of adisplay 306. Images of a participant 308, for example, are captured bythe cameras 302 and 304 and processed to create a single image of theparticipant 308 which appears to be captured by a single virtual camera310 for viewing by another person at a different location. However, animage captured in this manner typically suffers from processingartifacts, especially when the captured views are at a very differentangle from the intended virtual view, as would be the case with aparticipant located close to a large screen. This system also fails tocapture hand gestures near, or drawing on, the screen surface.

It is desirable to have visual-collaborative systems that project imageswithout interfering with and diminishing the quality of the imagessimultaneously captured by a camera.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1-3 show schematic representations of systems configured toproject images without interfering with images captured by a camera.

FIG. 4 shows a schematic representation of a first visual-collaborativesystem configured in accordance with one or more embodiments of thepresent invention.

FIG. 5 shows a plot of exemplary wavelength ranges over which twofilters transmit light in accordance with one or more embodiments of thepresent invention.

FIG. 6 shows a schematic representation of a second visual-collaborativesystem configured in accordance with one or more embodiments of thepresent invention.

FIG. 7A shows a schematic representation of a third visual-collaborativesystem configured in accordance with one or more embodiments of thepresent invention.

FIG. 7B shows two color wheels configured in accordance with one or moreembodiments of the present invention.

FIG. 7C shows plots of exemplary wavelength ranges over which twofilters transmit light in accordance with one or more embodiments of thepresent invention

FIG. 8 shows a schematic representation of a sixth visual-collaborativesystem configured in accordance with one or more embodiments of thepresent invention.

FIG. 9 shows a camera positioned at approximately eye level to aparticipant in accordance with one or more embodiments of the presentinvention.

FIG. 10 shows a schematic representation of a seventhvisual-collaborative system configured in accordance with one or moreembodiments of the present invention.

FIG. 11 shows a schematic representation of an eightvisual-collaborative system configured in accordance with one or moreembodiments of the present invention.

FIG. 12 shows a schematic representation of a ninth visual-collaborativesystem configured in accordance with one or more embodiments of thepresent invention.

FIGS. 13A-13B show a schematic representation of a tenthvisual-collaborative system configured in accordance with one or moreembodiments of the present invention

FIG. 14 shows a schematic representation of a visual-collaborativesystem configured in accordance with one or more embodiments of thepresent invention.

FIGS. 15A-15B show mixing of video content from two or more imagesources in accordance with one or more embodiments of the presentinvention.

FIG. 16 shows a top view of an interaction perceived by avideo-collaboration participant in accordance with one or moreembodiments of the present invention.

FIGS. 17A-17C each show an arrangement of video-collaborationparticipants located at a different site in accordance with embodimentsof the present invention.

FIG. 18 shows an example of generating a shadow in overlapping images ofparticipants in accordance with one or more embodiments of the presentinvention.

FIGS. 19A-19C show visual cues used to identify overlapping remoteparticipants in accordance with one or more embodiments of the presentinvention.

FIGS. 19E-19F show participants repositioned to avoid overlap inaccordance with one or more embodiments of the present invention.

FIGS. 20A-20B each show an arrangement of two video conferenceparticipants located at different sites in accordance with embodimentsof the present invention.

FIGS. 21A-21B show a participant repositioned to avoid overlap with ashared content window in accordance with one or more embodiments of thepresent invention.

FIGS. 22A-22B show a visual-collaborative system configured with atouchscreen and operated in accordance with one or more embodiments ofthe present invention.

FIGS. 23A-23C show visual collaboration between participants in a mirrormode in accordance with one or more embodiments of the presentinvention.

FIGS. 24A-24B show examples of mirrored and unmirrored images inaccordance with one or more embodiments of the present invention.

FIG. 25 shows an example of a private window displayed on a displayscreen in accordance with one or more embodiments of the presentinvention.

FIG. 26 shows a flow diagram of a method for establishingvisual-collaborative interaction in accordance with one or moreembodiments of the present invention.

FIG. 27 shows a schematic representation of a computing deviceconfigured in accordance with one or more embodiments of the presentinvention.

DETAILED DESCRIPTION

Embodiments of the present invention are directed tovisual-collaborative systems and methods enabling geographicallydistributed groups to engage in face-to-face, interactive collaborativevideo conferences. The systems include a projection display screen thatenables cameras to capture images and depth information of the localobjects through the display screen and send the images to a remote site.In addition, the display screen can be used to simultaneously displayimages from the remote site.

FIG. 4 shows a schematic representation of a visual-collaborative system400 configured in accordance with one or more embodiments of the presentinvention. The system 400 comprises a display screen 402, a camera 404,and a projector 406 and includes a filter A disposed between the cameralens 408 and the screen 402 and a filter B disposed between theprojector lens 412 and the screen 402. The camera lens 408 and projectorlens 412 are positioned to face the same first surface 410 of thedisplay screen 402. In the embodiments described in FIGS. 4-9, thescreen 402 is a rear projection display screen. However, the rearprojection implementation shown is for purposes of example only and thescreen 402 may also be a front projection display screen. A frontprojection implementation is shown in FIGS. 10-13.

Referring to FIG. 4, the screen 402 is a rear projection display screencomprising a screen material that diffuses light striking the firstsurface 410 within a first range of angles. The projector 406 ispositioned to project images onto the first surface 410 within the firstrange of angles. A participant 414 facing the outer second surface 416of the screen 402 sees the images projected onto the screen 402 from theprojector 406. The screen 402 is also configured to transmit lightscattered from objects facing the second surface 416. In other words,the camera lens 408 is positioned to face the first surface 410 so thatlight scattered off of objects facing the second surface 416 passthrough the display screen and is captured as images of the objects bythe camera 404.

In certain embodiments, the display screen 402 comprises a relativelylow concentration of diffusing particles embedded within a transparentscreen medium. The low concentration of diffusing particles allows acamera 404 to capture an image through the screen (providing the subjectis well lit), while diffusing enough of the light from the projector 406to form an image on the screen. In other embodiments, the display screen402 can be a holographic film that has been configured to accept lightfrom the projector 406 within a first range of angles and transmit lightthat is visible to local participant 1504 within a different range ofviewing angles. The holographic film is otherwise transparent. In bothembodiments, light projected onto the first surface 410 within the firstrange of angles can be observed by viewing the second surface 416, butlight striking the second surface 416 is transmitted through the screen402 to the camera. However, in both embodiments the camera 404 alsocaptures light from the projector 406 diffused or scattered off thefirst surface 410.

In order to prevent ambient light from striking the first surface 410 ofthe screen 402 and reducing the contrast of the projected and capturedimages, the system 400 may also include a housing 418 enclosing thecamera 404 and projector 406. The housing 418 is configured with anopening enclosing the boundaries of the screen 402 and is configured sothat light can only enter and exit the housing 418 through the screen402.

As shown in FIG. 4, filters A and B are positioned so that light outputfrom the projector 406 passes through filter B before striking the firstsurface 410 and light captured by the camera 404 passes through filterA. The filters A and B are configured to prevent light produced by theprojector 406 and scattered or diffused from the screen 402 frominterfering with light transmitted through the screen 402 and capturedby the camera 404. In one embodiment, this is achieved usingcomplementary filters to block different components of light. In oneembodiment, filter A passes through light that would be blocked byfilter B. Similarly, filter B passes light that would be blocked byfilter A. In this way, light from the projector 406 that is diffused orscattered off the first surface may be blocked.

This implementation (filter A passing light blocked by filter B andfilter B passing light blocked by filter A) is implemented in FIG. 4where the camera system includes a first filter (filter A) that isdisposed between the camera and the first surface of the display screen.Filter A passes the light received by the camera, except for the lightproduced by the projector (which it blocks). A second filter (filter B)disposed between the light source of the projector and the projectionsurface of the display screen, wherein the second filter passes lightoutput by the projector that is blocked by the first filter.

If the material used for the display screen 402 maintains polarizationof scattered light, and if the projectors used are the type which resultin no polarization of the light output from the projectors, thenpolarized filters may be used. In one embodiment, the complementaryfilters A and B are polarizing filters, where polarizing filter A has afirst direction of orientation that is different than the direction oforientation of polarizing filter B. In one embodiment, the filters arecircularly polarized, where the polarization for one filter is rightcircularly polarized and the polarization for the other filter is leftcircularly polarized. In one embodiment, the two filters are polarizedlinearly. In this embodiment, one filter is polarized horizontally whilethe other filter is polarized vertically.

Although the term blocked is used throughout the application, it isrealized that in some cases a filter might not block 100% of the lightof the complementary filter so that the filters are completelynon-overlapping. However, when the filters are non-overlapping, the bestperformance is typically achieved. For example, in the embodiment wherethe filters are linearly polarized with one filter (assume for purposesof example filter A) is polarized horizontally and the other filter(filter B) is polarized vertically, preferably, the direction oforientation of the filters is orthogonal to each other. In thisimplementation, the filters are non-overlapping and filter A blockslight that would not be blocked by filter B and filter B blocks lightthat would not be blocked by filter A. Although orientations other thana 90 degree orthogonal positioning may be used, this is not desirablesince as the orientation of the two filters moves further away from it'sorthogonal positioning, relative to each other, the further the systemperformance is decreased.

For purposes of example, assume that filter A is positioned at an 88degree angle relative to filter B (as opposed to the preferred 90 degreepositioning.) Although the filters are not completely non-overlapping,typically the filter arrangement would still provide a configurationthat would substantially block light from the complementary filter suchthat performance is not noticeably degraded to the participant (ascompared to the 90 degree orthogonal positioning). The degree to whichthe images are visually degraded is to some extent a function of themedia content and the environment (brightness, etc) of the participants.For example, if the media content includes a black and whitecheckerboard image (high brightness for white image and high contrast),an 88 degree relative positioning may not be sufficientlynon-overlapping to provide an image that is not noticeably degraded. Incontrast, if the media content is relatively dark compared to thecheckerboard content or the participant is an a low light environmentfor example, an 88 degree relative positioning of the filter may providelittle if any noticeable degradation by the participant. Thus for thiscase, the 88 degree relative position which substantially blocks (butnot completely blocks) the light produced by the projector results inminimum degradation of performance. Thus “block” and “substantiallyblocked” may be used interchangeable as long as difference in blockingresults in visual degradation that is either minimal or not apparent tothe participant. Light that is “substantially blocked” by a filter maycorrespondingly be “substantially transmitted” by it's complementaryfilter.

As previously noted, it is desirable for the filters A and B to beconfigured to prevent light produced by the projector and scattered ordiffused from the screen 402 from interfering with light transmittedthrough the screen 402 and captured by the camera 404. In the embodimentpreviously described, this is accomplished using a first type of filter,a polarized filter. However, other types of filters may be used. In analternative embodiment, this can be achieved using a second type offilter, a wavelength division filter.

In particular, filter B can be configured to transmit a first set ofwavelengths ranges that when combined create the visual sensation of amuch broader range of colors in projecting images on the display screen402, and filter A can be configured to transmit a second set ofwavelength ranges that are different from the first set of wavelengthranges. The second set of wavelength ranges can also be used to createthe visual sensation of a much broader range of colors. In other words,filter A is configured and positioned to block the wavelength rangesthat are used to create images on the display screen 402 from enteringthe camera lens 408. Even though the wavelength ranges used to produceimages viewed by local participant 1504 are different from thewavelengths of light used to capture images by the camera 404, theprojector 406 can still use the colors transmitted through filter B toproject full color images and light transmitted through filter A andcaptured by the camera 404 can still be used to record and send fullcolor images. It is the component wavelengths of the light used toproject and capture the full color images that are prevented frominterfering. Similar to the descriptions with respect to polarizedfilters, wavelength division filters may not completely benon-overlapping so that a filter may substantially block a set ofwavelength ranges.

FIG. 5 shows exemplary plots 502 and 504 of wavelength ranges over whichfilters A and B, respectively, can be configured to transmit light inaccordance with one or more embodiments of the present invention. Axis506 represents the range of wavelengths comprising the visual spectrum.Axis 508 represents intensities of light transmitted through filters Aand B, respectively. As shown in FIG. 5, the red, green and blueportions of the spectrum are each split into two halves with curves511-513 representing relatively shorter wavelength rangers of the red,green, and blue portions of visible spectrum transmitted through filterA and curves 515-517 representing relatively longer wavelength ranges ofthe red, green, and blue portions of visible spectrum transmittedthrough filter B. As shown in FIG. 5, filters A and B do not transmitthe same wavelength ranges of the red, green, and blue portions of thevisible spectrum. In particular, filter A is configured to transmitshorter wavelength ranges of the red, green, and blue portions of thevisible spectrum, and substantially block the longer wavelength rangesof the red, green, and blue portions of the spectrum. In contrast,filter B is configured to transmit the longer wavelength ranges of thered, green, and blue portions of the visible spectrum and substantiallyblock the short wavelength ranges of the red, green, and blue portionsof the visible spectrum. Both sets of red, green, and blue wavelengthscan be treated as primary colors that can be combined to produce a fullrange of colors in projecting images on the display screen 402 andcapturing images through the display screen 402. Thus, the combinationof filters A and B effectively block the light used to project colorimages on the display screen 402 form being back scattered andinterfering with the color images captured by the camera 404.

In other embodiments, operation of the filters A and B can be reversed.In other words, filter A can transmit the longer wavelength ranges ofthe red, green, and blue portions of the visual spectrum while filter Btransmits the shorter wavelength ranges of the red, green, and blueportions of the visible spectrum.

FIG. 6 shows a visual-collaborative system 600 configured in accordancewith one or more embodiments of the present invention. The system 600 isnearly identical to the system 400 except filter B and the projector 406are replaced with a single projector 602 configured to project colorimages using wavelength ranges that are blocked by filter A. Forexample, the projector 602 can be a conventional projector using threemicrodisplays and color splitting optics that send red, green and bluelight from the projector bulb to the corresponding display. Themicrodisplays can be well-known liquid crystal display (“LCD”), liquidcrystal on silicon (“LCoS”), or digital-micromirror device (“DMD”)technologies. In such a system, the functionality of filter B can beincorporated into the color splitting optics within the projector 602.Filter A is configured to transmit wavelength ranges other than thewavelengths reflected by the color splitter, as described above withreference to FIG. 5. For example, the internal color splitter can be aseries of dichroic mirrors that each reflects one of the primary colorsto a separate microdisplay, while passing other wavelengths of light.Each reflected color is modulated by the corresponding microdisplay, andthe colors are recombined to produce images that are projected onto thefirst surface 410. Each microdisplay provides pixelized control of theintensity of one color. The colors not reflected by the color splitterare discarded. For example, in order to produce a red object, themicrodisplays corresponding to projecting green and blue light areoperated to block green and blue light from passing through theprojector 602 lens.

In other embodiments, the lamp producing white light and the internalcolor splitter of the projector 602 can be replaced by separate lasers,each laser generating a narrow wavelength range of light that whencombined with appropriate intensities produce a full range of colors.For example, the lamp and internal color splitter can be replaced bythree lasers, each laser generating one of the three primary colors,red, green, and blue. Each color produced by a different laser passesthrough a corresponding LCD or is reflected off of a corresponding LCoSand the colors are recombined within the projector 602 to project fullcolor images onto the first surface 410. Note that the use of arelatively narrow set of wavelengths at the projector allows thecomplementary set of wavelengths passed by filter A to be relativelybroader, allowing more light into the captured image.

In other embodiments the function of filter A could be incorporated intothe camera optics. For example the color filter mosaic that forms partof a camera's image sensor could be selected to pass only selectedwavelengths.

FIG. 7A shows a visual-collaborative system 700 configured in accordancewith one or more embodiments of the present invention. The system 700 isnearly identical to the system 400 except filter B and the projector 406are replaced with a sequential color projector 702. An example of such aprojector is a “DMD projector” that includes a single digitalmicromirror device and a color wheel filter B comprising red, green, andblue segments. The color wheel filter B spins between a lamp and theDMD, sequentially adding red, green, and blue light to the imagedisplayed by the projector 702. Also, filter A is replaced by a secondcolor wheel filter A which contains filters that transmit complementarycolors to those of filter B. For example, as shown in FIG. 7B, the colorwheel filter A can use cyan, yellow, and magenta transparent colorpanels to sequentially block the color being projected through the colorwheel filter A. Color wheel filters A and B can be synchronized so thatwhen the color wheel filter A transmits one color the color wheel filterB transmits a complementary color. For example, when the red panel ofthe color wheel filter B passes between the lamp and the DMD of theprojector 702, the color red is projected onto the screen 402 while thecyan panel of the color wheel filter A covers the lens 408 enabling thecamera 404 to capture only green and blue light and ignore the projectedred light.

FIG. 7C shows exemplary plots 704-706 of wavelength ranges over whichcolor wheel filters A and B, respectively, can be operated to transmitlight in accordance with one or more embodiments of the presentinvention. Plot 704 shows that at a first time T₁, filter B passes adifferent range of wavelengths than filter A. Plot 705 shows that at alater second time T₂, filter B passes a range of wavelengths sandwichedbetween two different wavelength ranges passed by filter A. Plot 706shows that at a later time T₃, filter B again passes a different rangeof wavelengths than filter A. In other words, plots 704-706 reveal thatat any given time, filters A and B are operated to pass differentwavelength ranges. Plots 704-706 also reveal that filters A and B can beoperated to pass wavelengths over the same wavelength ranges, but not atthe same time.

In still other embodiments, the housing 418 can include fully reflectivemirrors that reflect projected images onto a display screen within therange of angles for which the screen is diffusive. FIG. 8 shows avisual-collaborative system 800 configured in accordance with one ormore embodiments of the present invention. The system 800 is nearlyidentical to the system 400 except mirrors 802 and 804 are included toreflect images produced by the projector 406 onto a display screen 806within a range of angles for which the screen 806 is diffusive.

The visual-collaborative systems described above with reference to FIGS.4-8 can be used in interactive video conferencing. The camera 404 andprojector 406 can be positioned so that the display screen 402 acts as awindow to a remote site. This can be accomplished by positioning thecamera 404 at approximately eye level to local participant 1504 facingthe second surface 416 and at approximately the same distance localparticipant 1504 would feel comfortable standing away from the screen.FIG. 9 shows the camera 404 positioned at approximately eye level tolocal participant 1504 in accordance with one or more embodiments of thepresent invention. As a result, local participant 1504 appearsface-to-face with a second participant represented by dashed-line FIG.902 located at a remote site. The second participant 902 and localparticipant 1504 can engage in an interactive, virtual, face-to-faceconversation with the display screen 402 serving as a window throughwhich the second participant and local participant 1504 can clearly seeeach other.

FIG. 10 shows a schematic representation of a seventhvisual-collaborative system configured in accordance with one or moreembodiments of the present invention. As previously stated, FIGS. 4-9are shown implemented using a rear-projection configuration. Thevisual-collaborative systems shown in FIGS. 10-13 are implemented usinga front-projection implementation. The systems are similar in that inboth rear and front projection systems project images onto a projectionsurface where the projected image is visible on the second surface ofthe display screen. However, the position of the camera and possibly thematerials used for the display screen or the display screenconfiguration may be different.

Similar to the implementation shown in FIG. 4, the embodiment shown inFIG. 10 includes a display screen 402, a camera lens 404, and aprojector 406. However, instead of being positioned behind or in therear of the screen (relative to local participant 1504), the projector406 in FIG. 10 is positioned in front of the display screen. Theprojector 406 projects an image onto a projecting surface 415. In thiscase, the projection surface 415 is the second surface of the displayscreen 102. The projected image is diffusely reflected off the secondsurface and can be observed by viewing the second surface.

In FIG. 10 the display screen 402 is a front-projection display screen.In one embodiment, the display screen 402 is comprised of a partiallydiffusing material that diffuses light striking it within a first andsecond range of angles. A participant 414 facing the outer secondsurface 416 of the screen 402 sees the images projected onto the screen402 from the projector 406. Similar to the embodiments described inFIGS. 4-9, the screen is configured to transmit light scattered fromobjects facing the second surface 416. In other words, the lens of thecamera is positioned to face the first surface 410 so that light fromobjects facing the second surface 416 pass through the display screenand is captured by the camera 404.

In one embodiment, the display screen is comprised of a material thathas a relatively low concentration of diffusing particles embeddedwithin a transparent screen medium. The low concentration of diffusingparticles allows a camera 404 to capture an image through the screen(providing the subject is well lit), while it diffuses enough of thelight from the projector 406 to form an image on the screen. In analternative embodiment, the display screen 402 is comprised of aholographic film that has been configured to accept light from theprojector 406 within a first range of angles and reflect light that isvisible to local participant 1504 within a different range of viewingangles. In some cases, the screen's partially diffusing material may nothave sufficient reflective properties to reflect the projected imagefrom the second surface of the display screen. In this case, the displayscreen includes a half silvered material (not shown) may be positioneddirectly behind and preferably in contact with the first surface of thedisplay screen. The half silvered mirror will allow transmission oflight through the display screen while enhancing the reflectivity of theholographic film.

In the front projection screen embodiment, the light projected onto thesecond surface within the first range of angles is diffused by thescreen and can be observed by viewing the second surface 416 and lightscattered off of objects facing the second surface are transmittedthrough the display screen to the camera. In the front projectionembodiment, light from the projector that is transmitted through thedisplay screen can degrade the performance of the system. In order tominimize this degradation, a filter A disposed between the camera andthe first surface of the display screen is used to block the lightreceived by the camera that is produced by the projector. In addition,in the preferred embodiment a filter B disposed between the projector'slight source and the projection surface (in this case the secondsurface) where the second filter passes light output by the projectorthat is blocked by the first filter.

FIG. 11 shows a schematic representation of an eighthvisual-collaborative system configured in accordance with one or moreembodiments of the present invention. The implementation of theembodiment shown in FIG. 11, is similar to that of FIG. 10, except forthe camera placement and the addition of a mirror 480. The mirror 480 isa completely reflective mirror with an opening 482 for the placement ofthe filter B. Although the completely reflective mirror improves theprojection image, light cannot pass through it. Thus, the camera'sposition changes. In one embodiment, the camera is positioned so that itis in physical contact with the display system fitter B. Since thecamera is not a distance away from the display screen, any writings onthe display screen such as is shown in FIG. 14, are not easily viewable.

FIG. 12 shows a schematic representation of a ninth visual-collaborativesystem configured in accordance with one or more embodiments of thepresent invention. The implementation of the embodiment shown in FIG. 12is similar to that shown in FIG. 10. However, instead of the displayscreen being comprised of a partially diffusing material, the displayscreen is comprised of standard front-projection screen material. Thereplacement of the display screen with standard projection screenmaterial decreases costs. However, because the standard projectionscreen material does not transmit light, the implementation of acollaborative board as shown in FIGS. 14A and 14B is not feasible usingthis configuration. In the embodiment shown in FIG. 13A-13B, the displayscreen includes an opening. Similar to the embodiment shown in FIG. 12,a filter A is positioned so that the filter covers the opening. A camerais positioned so that it's lens abuts the filters so that light receivedby the camera is filtered by filter A.

FIGS. 13A-13B shows a schematic representation of a tenthvisual-collaborative system configured in accordance with one or moreembodiments of the present invention. The representation in FIGS.13A-13B shows a rear projection screen implementation which is capableof projecting and capturing stereoscopic 3D images. Although theembodiments shown in FIGS. 13A-13B show a rear projection screenimplementation, alternatively the embodiments could be used in a frontprojection screen implementation. In both the rear projection screen andfront projection screen implementations, instead of a single projector,two projectors, a right projector and a left projector are used.Although FIGS. 13A and 13B show two cameras, a right camera and a leftcamera, alternatively a single camera may be used. In the case where twocameras and two projectors are used, the remote user and the projectedimage will both appear in 3D. In the embodiment where a single camera isused, the remote user will no longer appear in 3D, however, theprojected image will still appear in 3D.

Similar to the embodiments described with respect to FIGS. 4-11, lightproduced from each projector is blocked by the filters that pass lightreceived by each camera. For the 3D implementation to work, the screenmaterial for the embodiments shown in FIGS. 13A-B needs to bepolarizing-preserving material. In the embodiment shown in FIG. 13A,each camera has an identical wavelength division filter. For theprojector, two different filters (a polarizing filter and a wavelengthdivision filter) are used for each projector. For simplificationpurposes, the projectors used in the described implementation are thetype which result in no polarization of the light output from theprojectors.

In the embodiment shown in FIG. 13A, the two wavelength division filtersA are identical. The two polarizing filters are of the same type. Forexample, in one embodiment, the two polarizing filters are circularlypolarized filters where one filters is a right circularly polarizedfilter and the other filter is left circularly polarized filter. Inanother embodiment, the polarized filters are linearly polarized wherethe two polarizing filters are preferably orthogonal to each other. Forexample in one embodiment, for the left projector, a 45 degreepolarizing filter is used for the polarizing filter L and a wavelengthdivision color filter is used for WD filter A. For the right projector,a −45 degree polarizing filter is used for polarizing filter R and awavelength division color filter is used for WD filter A. The twowavelength division color filters used for the Right Projector and theLeft Projector should be identical. In the embodiment shown in FIG. 13A,the 3D image can be seen using L&R polarizing glasses.

In the embodiment shown in FIG. 13B, instead of the filters for thecameras being identical wavelength division filters, they are identicalpolarizing filters B. In the embodiment shown in FIG. 13B, again eachprojector has two corresponding different filters (a polarizing filterand a wavelength division filter). Again for simplification purposes,the projectors used in the described implementation are the type whichresult in no polarization of the light output from the projectors.

In the embodiment shown in FIG. 13B, the two filters used in conjunctionwith the projectors are wavelength division filters that block differentcomponents of light. The polarizing filters used in conjunction with theprojectors are of the same type. In the embodiment shown in FIG. 13B,the 3D image can be seen using wavelength division L&R glasses.

Embodiments of the present invention include using depth information inorder to determine the relative position of each participant, obtaininformation regarding the relative location of objects, or obtaininformation about the placement of a participant's hands. In certainembodiments, depth information can be collected by processing the imagescollected from the left and right cameras 404 described above withreference to FIGS. 13A and 13B. In other words, the left and rightcameras 404 can be stereo cameras oriented to provide three-dimensionalstereo images of the participants and objects facing the display screen402.

In other embodiments, a three-dimensional, time-of-flight camera, alsocalled a depth camera, can be included in the visual-collaborativesystem in order to provide depth information regarding the position ofthe participant and objects placed in front of the display screen 402.FIG. 14 shows a schematic representation of a visual-collaborativesystem 1400 configured in accordance with one or more embodiments of thepresent invention. The system 1400 is similar to thevisual-collaborative system 400 shown in FIG. 4 except the system 1400includes a depth camera 1402. The depth camera 1402 is an imaging systemthat creates distance data based on the time-of-flight principle. Thedepth camera 1402 illuminates a scene by generating short light pulses,such as infrared light pulses, that pass throught the screen 402. Thedepth camera 1402 includes sensors that measure the time elapsed forreflected infrared light pulses to return to the depth camera throughtthe screen 402. Each pixel of a digital image produced by the depthcamera 1402 includes depth information that can be correlated with theimages collected by the camera 404 and processed to separate or visuallyenhance objects based on the object's distance from the depth camera.

For the sake of brevity the depth camera 1402 is described as beingincorporated into the visual-collaborative system 400, but embodimentsof the present invention are not so limited. In other embodiments, thedepth camera 1402 can be included in the other visual-collaborativesystems described above with reference to FIGS. 8-12.

Note that in the follow discussion the terms “local participant” and“remote participant” are relative terms used to describe participantstaking part in a video conference using the visual-collaborative systemsdescribed herein. A participant interacting with another participantlocated at another site and via a display screen is referred to as alocal participant, and the participant displayed on the localparticipant's display screen is referred to as a remote participant. Forexample, consider a first participant located at a first site and asecond participant located at a second site. The first participant isreferred to as a local participant and the second participant isreferred to a remote participant when describing embodiments of thepresent invention from the site or position of the first participant.

In mixing video content projected onto a local participant's displayscreen, depth information provided by the remote participant can be usedto visually enhance or distinguish images of objects located closer tothe remote participant's display screen and suppress or remove entirelyimages of objects located farther from the remote participant's displayscreen. First consider simple mixing of two objects displayed on thedisplay screen 402 to a local participant. FIGS. 15A-15B show simplemixing of video content from two or more image sources in accordancewith one or more embodiments of the present invention. A shared contentwindow 1502 is projected onto the display screen 402 of a localparticipant 1504 and is displayed in an analogous manner for a remoteparticipant 1506 located at a remote site. Local participant 1504 andremote participant 1506 both see the same document or images displayedwithin the shared content window, and as shown in the example of FIG.15A, remote participant 1506 is pointing at a location within the window1502. FIG. 15B shows a top view of the interaction perceived by localparticipant 1504. Local participant 1504 perceives the remoteparticipant 1506 as being located behind the screen 402 and pointing toa location within the window 1502. The image of the window 1502 and theportion of the image of remote participant 1506 projected onto thewindow 1502 are mixed, which can obfuscate the content of the window1502 for local participant 1504.

In accordance with other embodiments of the present invention, mixing oftwo or more images based on depth information can be used to visuallysuppress or remove objects or portions of objects located within theborders of the window 1502 in order to limit obfuscation due to videomixing. FIG. 16 shows a top view of the interaction perceived by localparticipant 1504 when video content from two or more image sources ismixed based on depth information in accordance with one or moreembodiments of the present invention. Video of remote participant 1506and depth information captured at the remote participant's site can beprocessed together in projecting both the window 1502 and remoteparticipant 1506 on the display 402. The depth information is used tovary the opacity of the remote participant's image within the borders ofthe window 1502 based on the remote participant's distance from theremote participant's depth camera or display screen. As shown in the topview of FIG. 16, the window 1502 is displayed on the screen 402. Localparticipant 1504 perceives remote participant 1506 as standing behindthe screen 402 pointing at the window 1502, but the opacity of thewindow 1502 is segmented according to remote participant's 1506perceived distance from the window 1502. To local participant 1504,images of objects, such as the remote participant's hand, located withina first distance 1602 are nearly fully reproduced and mixed with thecontent of the window 1502. In other words, local participant 1504 seesthe remote participant's hand faithfully reproduced within the border ofthe window 1502 and mixed with the content of the window 1502. The imageof the window 1502 and objects projected behind the window 1502 areprocessed so that to local participant 1504 the opacity increases forobjects placed within a second distance 1603 from the display screen,and the opacity is further increased for objects placed within a thirddistance 1604 from the display screen. Objects located beyond thecombined distances 1601-1604 can be nearly or totally removed fromwithin the border of the window 1502. Video processing methods toincrease opacity can include alpha-blending the object and window imageswith increasing weight given to the window image, decreasing the colorsaturation and/or contrast of the object image, and blurring of theobject image (removing high frequency components such as edges). Theabove described methods reduce the object image's interference with thewindow image.

Video-collaboration systems of the present invention can also be used toprovide video conferencing for three or more sites. FIGS. 17A-17C eachshow an arrangement of video conference participants each participantlocated at a different site in accordance with embodiments of thepresent invention. Each of FIGS. 17A-17C shows a top view of aparticipant facing a display screen displaying two other participantsparticipating in the same video conference. As shown in FIGS. 17A-17C,for the sake of convenience each screen appears angled in order toreveal how the participants appear to the local participant. Each siteis configured with a camera, depth camera, and projector (not shown) asdescribed above with reference to FIG. 14 or a pair of stereo cameras(not shown) as described above with reference to FIGS. 13A-13B. Thecamera images collected at each site are used to place the participantswithin the screens at the other sites. For example, in FIGS. 17A and 17Bparticipants A and B are located to the left of center of theirrespective screens 1702 and 1704, and in FIG. 17C participant C islocated to the right of center of the screen 1706. Thus, in the exampleof FIG. 17A, participant A sees participant B displayed on the left sideof the screen 1702 and sees participant. C displayed on the right sideof the screen 1702. In FIG. 17B, participant B sees participant Adisplayed on the left side of the screen 1704 and sees participant Cdisplayed on the right side of the screen 1704. In certain embodiments,when two or more locations are displayed on a screen, depth informationcan be used to suppress the background at each site. In otherembodiments, that background at each site can be suppressed even whenone location is being displayed. For example, returning to FIG. 17A,depth information collected at participant B's site and depthinformation collected at participant C's site can be used to suppressthe backgrounds captured at participant B's and participant C'srespective sites so that the participants 13 and C appear to participantA as being located at the same remote site.

However, because the participant's can change positions at theirrespective sites, there may be instances when the participants willappear to overlap during a video conference. As shown in the exampleFIG. 17C, because participants A and B are both located to the left ofcenter at their respective sites, the images of participants A and Bdisplayed on screen 1706 overlap and appear mixed. In certainembodiments, depth information may be used to allow one remoteparticipant to be seen as in front of another remote participant, but itmay be the case both remote participants are similar distances fromtheir respective depth cameras, in which case their images appear mixed.

Embodiments of the present invention include providing visual cues tooverlapping participants enabling the overlapping remote participants toreposition themselves. In particular, each site can send information tothe overlapping remote participants that can be used by the overlappingremote participants to reposition themselves. For example, thevisual-collaborative system operated by participant C, shown in FIG.17C, identifies the images of participants A and B as overlapping ormixing and sends information to the participants A and B enablingparticipants A and B to take appropriate action such as repositioningthemselves.

Constructing overlapping images that enable overlapping participants toreposition themselves can be accomplished as follows. FIG. 18 shows anexample of generating a shadow in overlapping images of participants inaccordance with one or more embodiments of the present invention. In theexample of FIG. 18, image 1802 of participant B is sent 1804 toparticipant A for viewing. A depth-based silhouette of participant B,S_(B), 1806 is captured using the depth camera at participant B's siteand the silhouette information is also sent to participant A. Atparticipant A's site, a depth-based silhouette of participant A, S_(A),1808 is also collected. At participant A's site, the two silhouette's SAand SB are used to define three regions represented in combineddepth-based silhouettes 1810. The first region corresponds toparticipant B's background, S _(B). The second region corresponds towhere participant A's silhouette overlaps with participant B'ssilhouette, S_(A)∩S_(B). The third region corresponds to whereparticipant B's silhouette does not overlap with participant A'ssilhouette, S _(A)∩S_(B). In order to generate the image of participantB shown on participant A's screen 1702, participant B's background S_(B) is suppressed, the portion of participant B's image correspondingto S _(A)∩S_(B) is unaltered, and the portion of participant B's imagecorresponding to S_(A)∩S_(B) is darkened. As a result, participant Asees a shadow 1812 on participant B's image. The same operation can beperformed at participant B's site so that participant B is also aware ofhis/her overlap with participant A.

FIGS. 19A-19C show shadows used to identify overlapping remoteparticipants in accordance with one or more embodiments of the presentinvention. These shadows can be created from the depth camera data asdescribed above. As shown in FIG. 19A, participant A sees their ownshadow displayed on the image of participant B and no shadow displayedon participant C making participant A aware of overlap with participantB but no overlap with participant C. As shown in FIG. 19B, participant Balso sees a shadow displayed on the image of participant A and no shadowdisplayed on participant C making participant B aware of overlap withparticipant A but no overlap with participant C. In FIG. 19C,participant C does not see a shadow displayed on either participant A orparticipant B. Thus, participant C knows that he/she does not overlapwith either participant A or B. Note that the shadows are also generatedto provide participants A and B with visual cues as to how they canadjust their positions with respect one another. For example, in FIG.19A, participant A sees the shadow displayed on participant B appearingprimarily on the left side of participant B's image indicating thatparticipant A is located farther from the center than participant B. InFIG. 19B, participant B sees the shadow displayed on participant Aappearing primarily on the right side of participant. A's imageindicating that participant B is located closer to the center thanparticipant. A.

FIGS. 19D-19F show participants repositioned to avoid overlap inaccordance with one or more embodiments of the present invention. InFIG. 19D, participant A has moved farther to the left of center, and inFIG. 19C, participant B has moved toward the center. Participants A andB response to the visual cues provided by the shadows enableparticipants A and B to readjust their positions accordingly so thatimages of participants A and B do not substantially overlap, as shown inFIG. 19F.

Embodiments of the present invention include providing visual cues to aparticipant that overlap with a shared content window displayed on boththe local participant's display screen and the remote participant'sdisplay screen. The visual cue enables the overlapping participant tochange positions. FIGS. 20A-20B each show an arrangement of two videoconference participants located at different sites in accordance withembodiments of the present invention. Each Figure shows a top view of aparticipant facing a display screen displaying the other participant anda shared content window 2002. Each site is configured with a camera,depth camera, and projector (not shown) as described above withreference to FIG. 14 or a pair of stereo cameras (not shown) asdescribed above with reference to FIGS. 13A-13B. As shown in FIG. 20A,participant B has moved behind the window 2002 so that participant A canonly see a portion of participant B's face. As shown in FIG. 20B, ashadow image of participant B is generated within the window atparticipant B's site enabling participant B to recognize thatparticipant B appears to have stepped behind the window 2002 fromparticipant A's position. Note that the location of the shadow placed onthe window 2002 provides participant B with a visual cue as to howparticipant B can adjust his/her position with respect to the locationof the window 2002. FIGS. 21A-21B show participant. B repositioned toavoid overlap with the window 2002 in accordance with one or moreembodiments of the present invention. In FIG. 21A, participant B hasmoved farther to the right and appears in FIG. 21B to have moved outfrom behind the window 2002.

Shadows generated from depth-based silhouettes can also be used as avisual cue in other situations where it is desirable to induce theparticipants to re-orient themselves with respect to the system. Forexample, they could be used to induce participants to align themselveswith respect to the system's cameras so as to create better eye-contact.

Note that in other embodiments other kinds of visual cues can be used.For example, arrows directing participants to move either left or rightcan be used as visual cues for instructing participants to changepositions.

FIG. 22A shows a visual-collaborative system 2200 configured with atouchscreen 2202 in accordance with one or more embodiments of thepresent invention. The system 2200 is similar to the system 1400 exceptthe display screen 402 includes a touchscreen 2202. The touchscreen 2202is configured to detect the presence and location of the localparticipant's contact within the display area of the touchscreen. Forexample, FIG. 22B shows a snapshot of local participant 1504 drawing, anare 2204 on the touchscreen 2202 using a stylus. The touchscreenidentifies the coordinate locations of the markings comprising the are2204 and projects the markings on the local participant's display screen402 as the local participant generates the are and simultaneously sendsthe coordinates of the are as it is being created to the remoteparticipant 1506 so that the are 2204 can be displayed on the displayscreen of the remote participant 1506. In other words, the localparticipant's markings are reproduced as if the local participant wasdrawing at a white board that both the local and the remote participantare standing in front of the touchscreen 2202 can be any suitabletouchscreen. For example, in certain embodiments, the touch screen 2202can be configured with an array of infrared light-emitting diodesdisposed along two adjacent bezel edges of the display screen 402.Photodetectors are disposed along the two opposite bezel edges. Thelight-emitting diode and photodetector pairs create a grid of lightbeams across the display. An object, such as the local participant'sfinger or stylus, that touches the display screen interrupts the lightbeams, causing a measured decrease in light at the correspondingphotodetectors. The measured photodetector outputs can be used to locatea contact-point coordinate. In other embodiments, the touchscreen 2202can be implemented as a capacitive touchscreen panel comprising aninsulator, such as glass, coated with a transparent conductor, such asindium tin oxide. Touching the surface of the touchscreen results in adistortion of the local electrostatic field, which is measured as achange in capacitance at a coordinate location within the touchscreen2202. In other embodiments, the touchscreen 2202 can be a resistivetouchscreen panel including two thin, metallic, electrically conductivelayers separated by a narrow gap. When the local participant's finger orstylus presses down on a point on the touchscreen's outer surface, thetwo metallic layers make contact at that point. The two panels behavelike a pair of voltage dividers with connected outputs creating a changein the electrical current which is registered as a contact event at aparticular coordinate location on the touchscreen.

Embodiments of the present invention are not limited to the threeexamples of touchscreens described above. For the sake of simplicity,the three kinds of touchscreens described above are included just tomention a few of the many different kinds of touchscreens that aresuitable for the visual-collaborative system 2200 described above withreference to FIG. 22.

Returning to FIG. 22B, in order to enhance the visibility of the marksproduced by the local participant 1504, the marks can be enhanced bysurrounding the edges of the marks with a dark shadow. For example, asshown in FIG. 22B, the are 2204 generated by local participant 1504appears on the display screen as a white curve surrounded by a darkershadow that can be seen by both the local participant 1504 and theremote participant 1506.

Visual collaboration embodiments of the present invention are typicallyperformed in a mirror mode. FIGS. 23A-23C show visual collaborationbetween local participant 1504 and remote participant 1506 in the mirrormode in accordance with one or more embodiments of the presentinvention. As shown in the top view of FIG. 23A, in the mirrored mode,the image of remote participant 1506 is displayed for the localparticipant so that the left side of the local participant is oppositethe left side of the remote participant and the right side of the localparticipant is opposite the right side of the remote participant. Themirrored mode also mirrors messages displayed on the display screens ofthe respective participants so the messages appear correctly oriented oneach participant's display screen. For example, as shown in FIG. 23B, acorrectly oriented message 2302 is displayed on the local participant'sdisplay screen 402 so that the local participant can read to themessage. The mirrored mode enables the same messages 2302 to bedisplayed on the remote participant's display screen 2304 with thecorrect orientation, as shown in FIG. 23C.

The visual-collaborative system 2200 enables participants to switchbetween the mirrored mode and an unmirrored mode. For example, as shownin FIG. 23B, the display screen 402 can be projected on the touchscreen2202 with a mirrored/unmirrored icon 2306 located in the corner of thedisplay screen 402. FIG. 24A shows an example of an unmirrored imagedisplayed on the display screen 402 in accordance with one or moreembodiments of the present invention. When the local participantcontacts the icon 2306, shown in FIG. 23B, the image displayed to localparticipant 1504 is unmirrored by flipping or rotating the image 180degrees about a vertical axis 2402 shown in FIG. 24A. Switching to theunmirrored mode reverses the orientation of the message 2302 so that themessage appears backwards to local participant 1504. As shown in the topview of FIG. 24B, in the unmirrored mode, the image of remoteparticipant 1506 is displayed so that the left side of remoteparticipant 1506 is opposite the right side of local participant 1504and the left side of local participant 1504 is opposite the right sideof remote participant 1506. Note that mirroring or unmirroring an imageis a local operation. In other words, although the image appearsunmirrored to the local participant 1504, the image displayed for theremote participant remains mirrored until the remote participantperforms the same operation.

Embodiments of the present invention include enabling participants tobring up one or more private windows within the display screen. The oneor more private windows can be viewed only by the local participant andcannot be viewed by the remote participants. FIG. 25 shows an example ofa private window 2502 displayed on the display screen 402 of localparticipant 1504 in accordance with one or more embodiments of thepresent invention. The private window 2502 cannot be viewed by theremote participant 1506. The private window can be an e-mail window, aconfidential document, a web browser, or any other graphic userinterface.

FIG. 26 shows a flow diagram of a method for establishing visualcollaborative interaction in accordance with one or more embodiments ofthe present invention. Steps 2601-2605 do not have to be completed inany particular order and can be performed at the same time. In step2601, images of one or more local participants are captured using one ormore cameras as described above with reference to FIGS. 4 and 6-14. Instep 2602, depth information of the one or more local participants iscollected using either stereo cameras or a depth camera as describedabove with reference to FIGS. 13 and 14. In step 2603, the images anddepth information associated with the one or more local participants istransmitted to one or more remote participants. In step 2604, the localparticipant receives images and depth information from the one or moreremote participants. In step 2605, the image of the one or more remoteparticipants is displayed on a display screen at the local participant'ssite based on the depth information as described above with reference toFIGS. 15-25.

In general, the methods employed to establish visual collaborationbetween a local participant and one or more remote participants can beimplemented on a computing device, such as a desktop computer, a laptop,or any other suitable computational device. FIG. 27 shows a schematicrepresentation of a computing device 2700 configured in accordance withone or more embodiments of the present invention. The device 2700includes one or more processors 2702, such as a central processing unit;a touchscreen interface 2704; one or more projectors interfaces 2706; anetwork interface 2708, such as a Local Area Network LAN, a wireless802.11x LAN, a 3G mobile WAN or a WiMax WAN; a camera system interface2710; and one or more computer-readable mediums 2712. Each of thesecomponents is operatively coupled to one or more buses 2714. Forexample, the bus 2714 can be an EISA, a PCI, a USB, a FireWire, a NuBus,or a PDS.

The computer readable medium 2712 can be any suitable medium thatparticipates in providing instructions to the processor 2702 forexecution. For example, the computer readable medium 2712 can benon-volatile media, such as an optical or a magnetic disk; volatilemedia, such as memory; and transmission media, such as coaxial cables,copper wire, and fiber optics. Transmission media can also take the formof acoustic, light, or radio frequency waves. The computer readablemedium 2712 can also store other software applications, including wordprocessors, browsers, email, Instant Messaging, media players, andtelephony software.

The computer-readable medium 2712 may also store an operating system2716, such as Mac OS®, Microsoft Windows®, Unix®, or Linux®; networkapplications 2718; and a collaboration application 2720. The operatingsystem 2716 can be multi-user, multiprocessing, multitasking,multithreading, real-time and the like. The operating system 2716 canalso perform basic tasks such as recognizing input from input devices,such as a keyboard or a keypad; sending output to the projectorinterface 2706; keeping track of files and directories on medium 2712;controlling peripheral devices, such as disk drives, printers, camerasystems; and managing traffic on the one or more buses 2714. The networkapplications 2718 include various components for establishing andmaintaining network connections, such as software for implementingcommunication protocols including TCP/IP, HTTP, Ethernet, USB, andFireWire.

The collaboration application 2720 provides various software componentsfor establishing visual collaboration with one or more remoteparticipants, as described above. In certain embodiments, some or all ofthe processes performed by the application 2720 can be integrated intothe operating system 2716. In certain embodiments, the processes can beat least partially implemented in digital electronic circuitry, or incomputer hardware, firmware, software, or in any combination thereof.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the invention.However, it will be apparent to one skilled in the art that the specificdetails are not required in order to practice the invention. Theforegoing descriptions of specific embodiments of the present inventionare presented for purposes of illustration and description. They are notintended to be exhaustive of or to limit the invention to the preciseforms disclosed. Obviously, many modifications and variations arepossible in view of the above teachings. The embodiments are shown anddescribed in order to best explain the principles of the invention andits practical applications, to thereby enable others skilled in the artto best utilize the invention and various embodiments with variousmodifications as are suited to the particular use contemplated. It isintended that the scope of the invention be defined by the followingclaims and their equivalents:

1. A method for establishing a collaborative interaction between a localparticipant and one or more remote participants, the method comprising:capturing images of the local participant in front of a display screen;collecting depth information of the local participant located in frontof the display screen; transmitting the images and depth information ofthe local participant to the one or more remote participants; receivingimages and depth information of the one or more remote participants; andprojecting the images of the one or more remote participants on thedisplay screen based on the depth information of the remoteparticipants; wherein projecting the images of the one or more remoteparticipants on the display screen based on the depth information of theremote participants further comprises: identifying two or moreoverlapping remote participants; and for each overlapping remoteparticipant, generating a visual cue appearing on the overlapping remoteparticipant's display screen, the visual cue including information thatenables the overlapping remote participant to change positions to avoidoverlapping with another remote participant.
 2. The method of claim 1wherein capturing the images of the local participant further comprisespositioning the display screen between the local participant and one ormore cameras so that the images of the local participant can be capturedthrough the display screen.
 3. The method of claim 1 wherein collectingdepth information of the local participants further comprises: capturingstereo images of the local participant through the display screen; andprocessing the stereo images to obtain the depth information of thelocal participant.
 4. The method of claim 1 wherein collecting depthinformation of the local participant further comprises positioning adepth camera to collect the depth information of the local participantthrough the display screen.
 5. The method of claim 1 wherein collectingthe depth information of the local participant further comprisesdetermining coordinate locations of the local participant image andposition in front of the display screen.
 6. The method of claim 1wherein projecting the images of the one or more remote participants onthe display screen based on the depth information of the remoteparticipants further comprises: displaying a shared content windowwithin the display screen, the window appearing in the display screen ofeach participant; and displaying the images of the one or more remoteparticipants to appear behind the window wherein the opacity of thewindow increases based on the depth information associated with the oneor more remote participants.
 7. The method of claim 6 wherein displayingthe images of the one or more remote participants to appear behind thewindow further comprises: decreasing the transparency of the window;reducing color saturation of the window; and blurring objects thatappear located, behind the window.
 8. The method of claim 1 wherein thevisual cue further comprises a shadow.
 9. The method of claim 1 furthercomprising: detecting contact made by the local participant on thedisplay screen; displaying markings corresponding to the contact on thedisplay screen; and transmitting the markings to the one or more remoteparticipants so that the markings can be displayed on associated displayscreens.
 10. The method of claim 9 wherein the markings include asurrounding relatively darker shadow.
 11. The method of claim 1 furthercomprising generating a private window within the display screen, theprivate window viewed exclusively by the local participant.
 12. A methodfor establishing a collaborative interaction between a local participantand one or more remote participants, the method comprising: capturingimages of the local participant in front of a display screen; collectingdepth information of the local participant located in front of thedisplay screen; transmitting the images and depth information of thelocal participant to the one or more remote participants; receivingimages and depth information of the one or more remote participants; andprojecting the images of the one or more remote participants on thedisplay screen based on the depth information of the remoteparticipants; wherein projecting the images of the one or more remoteparticipants on the display screen based on the depth information of theremote participants further comprises: identifying when the image of theone or more remote participants is projected behind a shared contentwindow displayed on the display screen; and generating a visual cuedirecting the one or more remote participants to change positions toavoid being projected behind the window.
 13. The method of claim 12wherein the visual cue further comprises a shadow.
 14. The method ofclaim 12 wherein capturing the images of the local participant furthercomprises positioning the display screen between the local participantand one or more cameras so that the images of the local participant canbe captured through the display screen.
 15. The method of claim 12wherein collecting depth information of the local participants furthercomprises: capturing stereo images of the local participant through thedisplay screen; and processing the stereo images to obtain the depthinformation of the local participant.
 16. The method of claim 12 whereincollecting depth information of the local participant further comprisespositioning a depth camera to collect the depth information of the localparticipant through the display screen.
 17. A method for establishing acollaborative interaction between a local participant and one or moreremote participants, the method comprising: capturing images of thelocal participant in front of a display screen; collecting depthinformation of the local participant located in front of the displayscreen; transmitting the images and depth information of the localparticipant to the one or more remote participants; receiving images anddepth information of the one or more remote participants; projecting theimages of the one or more remote participants on the display screenbased on the depth information of the remote participants, whereinprojecting the images of the one or more remote participants on thedisplay screen further comprises projecting mirror images of the one ormore remote participants displayed on the display screen; and detectinga signal made by the local participant, the signal unmirroring themirror image of the one or more participants displayed on the displayscreen.
 18. The method of claim 17 wherein capturing the images of thelocal participant further comprises positioning the display screenbetween the local participant and one or more cameras so that the imagesof the local participant can be captured through the display screen. 19.The method of claim 17 wherein collecting depth information of the localparticipants further comprises: capturing stereo images of the localparticipant through the display screen; and processing the stereo imagesto obtain the depth information of the local participant.
 20. The methodof claim 17 wherein collecting depth information of the localparticipant further comprises positioning a depth camera to collect thedepth information of the local participant through the display screen.